DOCUMENT RESUME 



ED 275 063 



EA 018 913 



AUTHOR 
TITLE 



INSTITUTION 

SPONS AGENCY 
PUB DATE 
GRANT 
NOTE 



PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



Peterson, Ken; Kauchak, Don 

Career Ladders and Teacher Incentives: The Utah 
Experiment. Part I: The Implementation. Final Report: 
Secretary's Discretionary Program Implementation 
Grant to Develop Teacher Incentive Structures. 
Utah Univ., Salt Lake City. Dept. of Educational 
Studies. 

Department of Education, Washington, DC. 
1 Jun 86 
G008410033 

76p.; A collaborative implementation effort among the 
University of Utah Department of Educational Studies 
and the Park City, Granite, Nebo, and Rich School 
Districts (Utah). For case studies, see EA 018 
914. 

Reports - Research/Technical (143) 
MF01/PC04 Plus Postage. 

♦Career Ladders; Educational Innovation; Elementary 
Secondary Education; *Employer Employee Relationship; 
Faculty Development; Mncentives; Inservice Teacher 
Education; Job Enrichment; Rewards; School Districts; 
Teacher Attitudes; *Teacher Evaluation; *Teacher 
Promotion 
IDENTIFIERS *Utah 

ABSTRACT 

This volume, the first of two reports on development 
of teacher incentive structures, focuses on implementation of a 
career ladder design and teacher evaluation experiment in four Utah 
school districts. The report has five sections. Section 1 describes 
study background and implementation of an evaluation-reward system 
that featured data collection by teachers. A panel of teachers, 
administrators, and parents made promotion decisions, which included 
a stipend and job enlargement opportunities. Career ladder committees 
conducted evaluations. Section 2 describes the districts 9 plans and 
design features compared with preexisting approaches. The promotion 
system was intended to be a substantial career development r;ove for 
teachers. Section 3 discusses evaluation, promotion, and planning 
outcomes. Section 4 presents conclusions and recommendations, 
including findings that: (1) teachers can control their evaluation; 
(2) good evaluation requires resources; (3) inservice is crucial; (4) 
teachers 1 competition for quality performance creates difficulties; 
(5) teachers select promotion pay over job enlargement; and (6) merit 
pay is a threat to career ladders' development. Section 5 consists of 
nine appendices; these include detailed descriptions of the career 
ladder plans and evaluation system, supporting information on legal 
issues and use of microcomputers, and a bibliography. (CJH) 
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IMPLEMENTATION p. I 
SECTION ONEs INTRODUCTION 
Activities funded by this grant were in one of two these* s 
(1) implementations of a career ladder design and teacher 
evaluation in four Dtah districts and (2) case studies of teacher 
perceptions of career ladder implentation . This report ( Final 
Report: Part I ) describes the implementation in the demonstration 
sites. The case studies are contained in the companion volume 
Final Report: Part II , 

Background 

In 1984 the Utah Legislature provided $15 million for local 
districts on a discretionary basis to design career ladders. The 
following year the annual allocation was raised to $30.7 million; 
for 1986-87 the amount is approximately $38 million. The basic 
ideas of career ladder development were to: 

a. create a multi-level compensation program for teachers, 

b. lengthen the contract year 

c. provide a "performance bonus" for outstanding teaching, 
and 

d. create an opportunity for some teachers to work year 
round. ( A Call to Action , 1983). 

Dnlike other states (such as Tennessee), the Dtah approach was to 

encourage local development,' rather than to implement a state-wide 

plan. The career ladders program, like those in other states, has 

faced controversy, but the political situation suggests a 

continuance of the program (Jones, 1986; Nelson, 1986). 

The Dtah Teacher Evaluation Project (Peterson, Kauchak & 

Driscoll, 1983) has functioned at the Dniversity of Utah, 

Department of Educational Studies, as a resource for districts, 

legislature and other state agencies. The Project has tested 

career ladder developments with the Park City School District. 
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This grant enabled collaboration to be extended to additional 
districts in Utah. 

Implementation of an Evaluation-Reward System 

The purpose of the implementation theme of this study was to 
put in place an innovative teacher evaluation and reward system in 
four different Utah districts. 

The innovative evaluation and reward system developed for 
this study is a dossier-promotion approach. The first 
distinguishing feature is that teachers are expected to collect a 
set of data from at least four perspectives (or "lines of 
evidence") which they assemble in a dossier. The dossier is much 
smaller than a portfolio of teacher work; it is limitec 1 to 
approximately 15 pages. The second feature is that the promotion 
decision is made by a panel of reviewers which is teacher 
dominated., but includes administrators and parents. The final 
important feature of the system is a promotion which includes a 
title, stipend of approximately $1500, and opportunities for job 
enlargement assignments. The system is fully described in 
Appendices A, B and C of this report. 

The Implementation 

The teacher evaluation and reward system was installed in 
four diverse Utah school districts. Implementation was done by 
providing advice, models, materials, and inservice to the 
districts through career ladder committees which governed design 
and practice. All systems were reviewed and modified by State 
Office of Education. The same basic model of evaluation and 
reward was presented to each of the four implementation districts, 
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IMPLEMENTATION p. 3 

but changed according to the decisions of local career ladder 
committees. 

In three districts the promotion system was the main 
"vertical" feature of the ladder program. Job enlargement 
components constituted the remaining part of the programs. Job 
enlargement ranged from participation in evaluation activities by 
six teachers in one district, to fully half the system in 
additional days or duties for teachers in another district. In 
the fourth district the system was a small part of the total 
career ladder system (10% by budget) and functioned as a merit pay 
component • The districts are described in Section Two of this 
report . 

Evaluation of Implementations 
The main evaluation was derived from the district career 
ladder committees for the subsequent year* Each district 
conducted an evaluation program on career ladder development. 
Committee decisions following this implementation ranged from 
keeping the system intact in two districts, modification in a 
third, and abandoning it all together in the fourth district. . The 
findings and continuation results are presented in Section Three 
of this report. 

This Report 

This report is divided into five sections. Section One is an 
introduction. Section Two describes the four districts, and 
variations in their plans. Section Three describes the 
evaluation , promotion and planning outcomes . Section Four 
presents conclusions and recommendations. A detailed description 
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of the evaluation and career ladder plan is included as Appendices 
A, B and C. other supporting intonation, including legal issues, 
use of microcomputers, and Bibliography, appears in appendices in 
the fifth section. 
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SECTION TWO I THE CAREER LADDER SYSTEM 
The career ladder design was implemented in fonr Utah school 
districts* Granite, Nebo, Park City (known as "Snow District" in 
the case studies), and Rich School Districts. 'She districts 
ranged in size from the largest in the state (Granite-with 3,271 
teachers) to Rich (29 teachers). The Park city system has run for 
two years; the other districts for 1985-86 school year. Figure 1 
presents a comparative description of the four districts. 

FIGURE 1: FOUR IMPLEMENTATION DISTRICTS 

* OP I TEACHERS PROMOTION 

DISTRICT TEACHERS DESCRIPTION PROMOTED STIPEND 

Granite 3,271 urban, suburban 509 $1000.* 

Park City 66 ski resort 47 $1500. 

Nebo 635 suburban 333 $1200. 

Rich 29 rural 19 $2100. 

♦merit pay instead of promotion 

Utah career ladders have vertical and horizontal features, as 
well as merit pay (Utah state office of Education, 1986). 
Vertical features are distinctions among teachers made according 
to performance or additional, duties. The distinctions usually are 
accompanied by stipends ranging from several hundred to several 
thousand dollars. Vertical distinctions typically are made for 
one year only, the promotion design which was installed in this 
project is unique in the state. Horizontal features are 
provisions for all teachers iu the district, such as additional 
preparation days. Merit pay. was included under a requirement for 
a "performance bonus" which by State Office of Education 
regulation was to be at least 10% of the total career ladder 
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program in each district. For the year of this implementation, a 
promotion system was considered by the State regulators to fulfill 
performance bonus requirements. 

The promotion system which was a key part of this 
implementation is intended to be a substantial career development 
move for teachers. Promotions permit acknowledgement of good 
practice, while not being tied to narrow indicator? of quality or 
periods of performance. A promotion essentially is a reward for 
demonstrated quality teaching. 

Numbers of teachers to be promoted in a career ladder system 
is an issue of contention. In the system implemented in this 
study, the first stage of promotion ("Associate Teacher") was 
anticipated to be appropriate for as many as 70% of the teachers 
in a district. Some critics take the viewpoint that promotion 
means outstanding, and should be reserved for a small elite group* 
Others take the view that any teacher who vorks well deserves the 
reward. The design included both points of view by making the 
first promotion generally available to all who could demonstrate 
through their dossier that they had performed with quality. The 
second stage of promotion to "Teacher Leader" is still under 
development and was not included in this implementation. 

Teacher evaluation systems for the new career ladders within 
the State vary, but most emphasise principal reports using a 
checklist. Two dominant approaches include a clinical supervision 
model — the Improving Teacher Competence (ITC), and a "research 
based" checklist of classroom behaviors — (AIM). The evaluation 
design for this implementation is a considerable departure from 
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current evaluation practice. For example, principal input is 
deamphasised. Mao, a wide range of measures is included; for 
example, teacher tests (National Teachers Examination) was used in 
the implementation districts, but nowhere else in Utah. Teacher 
control of evaluation lines of evidence and data is another 
provision that set the implementation districts apart from other 
career ladder approaches in the State. The dossier system was 
unique in the State. Finally, the use of panels of teachers, 
administrators, and parents was distinctive. 

Comparative Features of Basic Plan 
The basic career ladder for this implementation project was 
designed in light of preexisting approaches (Peterson, et al., 
1935). The plan differed from the 1970s Temple City experience in 
having a much more complicated teacher evaluation process, but 
less definition to teacher differentiation (Freiberg, 1984-5; 
English, 1984-5). The dossier in this project differs from the 
portfolio included in the Charlotte-Mecklenburg approach; teacher 
control of evaluation data and inclusion of parents on decision 
making panel are further distinctive features (Schlechty, Joslin, 
Leak & Hanes, 1984-5). Less reliance on classroom observation for 
teacher evaluation is characteristic of this model relative to the 
Tennessee career ladder plan (French, 1984-5) and the earlier 
approaches. 

The District Plans 
While the same career ladder model was presented to each 
district for implementation, actual operations were determined by 
local career ladder committees and regulated by the State Office 
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of Education. Consequently, the implementation* varied in the 
fcur districts. Tiro were very close tQ the original design. 
Figures 2 through 5 summarise the career ladder designs in each 
district. Additional descriptions are presented in Appendix c. 

The career ladder system for Park City was based on a 
"mainline" opportunity for all teachers to receive reward and 
recognition for classroom performance and for additional 
opportunities for some teachers, at any given time, to perform 
leadership duties for which they receive additional pay. The 
promotion stipend was set at $1500. per year for the five year 
duration of the promotion. 

Rich School District had goals in their design of (a) 
reducing the role of the principal in decision making and (b) 
minimizing extra work for extra pay. In addition, the District is 
fairly remote (northernmost in Utah) and has a problem with high 
turnover with beginning teachers. A promotion system that 
rewarded longevity was desired. Job enlargement in the District 
included two committees working on the career ladder itself 
(career ladder committee, and promotion panel), and several 
optional teacher initiated jobs. The promotion stipend was 
approximately $2200 per year for five years. 
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FIGURE 2: PARK CITY DISTRICT CAREER LADDER PLAN 

I. Number of Teachers* 66 

II. Allocation for career ladders: $101,280 

III. Vertical dimension: Three ladder levels (A) Certified 
Teacher, (B) Associate Teacher, and (C) Senior Teacher. 
Teachers are placed on these levels based on applications 
containing multiple lines of evidence that are judged by a 
promotion panel. Senior Teacher level will not be 
operational until the 1986-87 school year. 

IV. Job enlargement: several district prescribed or teacher 
generated options. District options include service on 
career ladder promotion and oversight committees and service 
as a curriculum committee chairperson or test developer. 
Teacher projects include a wide variety of options. 

V. Extended year: One mandatory day for all teachers to be used 
for parent conferences. 

VI. Advancement on career ladder: Teacher applies, submits an 

extensive dossier including four of a possible eight lines of 
evidence. Administrator report and pupil achievement data 
are required. Promotion panel reviews for acceptance or 
rejection? five of eight votes required. 

VI. Promotion or Bonus: Promotion for five year period. Annual 
stipend $1500. 
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FIGURE 3 i MEBO DISTRICT CAREER LADDER PLAN 

I. Number of Teachers t 635 

II. Allocation for career ladders: $1,129,282 

III. Vertical dimension: Four ladder levels. Level 3, Associate 
Educator, prepares a dossier that is reviewed by the 
promotion review board. Re-evaluation takes place during the 
third year at this level. Level 4, Senior Educator, has 
maintained the Associate Educator through two successive 
evaluations. 

IV. Job enlargement: Optional, by explication on all levels, as 
facilitators, promotion review board, and building level 
positions. Number of positions determined at the building 
level. Each building allocated $200 per PTE. Building 
committee works with principal for decisions. 

V. Extended year: Seven days for inservice training in 
essential elements of instruction, teacher preparation. 

VI. Advancement on career ladder: Promotion Wised on committee 
analysis of dossier. 

VI . Promotion or Bonus : Promotion for three years . Share of 
allocation to Performance Bonus, approximately $1100. 
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FIGURE 4 1 RICB DISTRICT CAREER LADDER PLAN 

X, Number of Teachers i 29 

II. Allocation for career ladders: $51,728 

III. Vertical dimensions Three ladder levels (A) Certified 
Teacher, (B) Associate Teacher, and (C) Senior Teacher. 
Teachers are placed on these levels based on applications 
containing multiple lines of evidence that are judged by a 
promotion panel. Senior Teacher will not be operational 
until 1986-87. 

IV. Job enlargement: Six additional service assignments: 

curriculum development, student teacher advisor, promotion 
review board, dossier oversight, mentor teacher, and special 
assignment. 

V. Extended year: Two additional days, one for dossier 

preparation and one for beginning of school year preparation, 

VI • Advancement on career ladder: review of extensive dossier 
containing at least 4 lines of evidence* Promotion board 
consists of teachers, administrators, and community 
representatives . 

VI. Promotion or Bonus: Share of Performance Bonus allocation 
(approximately $2000.). 
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FIGURE 5 I GRANITE DISTRICT CAREER LADDER PLAN 

I. Number of Teachers: 3,271 

II. Allocation for career ladders: $5,098,692 

III. Vertical dimension: teachers choose vertical dimension for 
one year, selected upon classroom evaluation, receive extra 
responsibilities and extended contract year. 

IV. Job enlargement: ten to twenty additional contract days for 
instruction related service to school or district. Service 
rendered during summer, before or after school. 

V. Extended year: Six and one half days total. One day prior 
to school-, One day at conclusion of school. Two days to 
students' instructional calendar. One half day for local 
school inservice training. 

VI. Advancement on career ladder: incentive personnel selection 

committee, at each school, consisting of 3 teachers, one PTA, 
and school principal. 

VI. Promotion or Bonus: Merit pay only — quotas, one year. Open 
to those with at least two full years with satisfactory 
evaluation. Applicants submit a dossier with at least four 
lines of evidence. Decisions made for regional panels. 
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Nebo School District also wanted to minimize principal 
decision Making and additional tasks. Their, horizontal component 
was the largest in the State, making seven additional working days 
available for every teacher. Nebo also had a greater number of 
job enlargement opportunities. While the evaluation and promotion 
plans were similiar to Rich, the horizontal provision meant that 
the promotion stipend was approximately half that of the Rich 
District. The duration of the Nebo promotion initially was set at 
three years to permit closer monitoring. 

Granite School District career ladder system was the most 
divergent of the four. The dossier system was used not for 
promotion, but for merit pay. Ten percent of the budget was 
allocated by State guideline for "performance bonus." The career 
ladder panel decided that the money would be limited to a given 
number of awards to make each $1000 for the one year. The 
resulting quotas meant that the decision panels made competitive 
judgements. The system included no long term vertical benefits. 
Incremental vs. Structural Changes for Teachers 
Career ladder plans may be designed to be lesser or greater 
changes for classroom teachers . So-called incremental changes 
have the advantage of slowly introducing improvements and allowing 
for development of new procedures. They consist of adding small 
improvements which promise to add up to something substantial. 
They have the disadvantage of not capturing the imagination and 
needs of teachers sufficiently to permit survival of the program 
(English, 1984-5) • Major structural changes for teachers may 
answer the challenges of the current educational reform movement 
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in this country, but they require a great deal of prior 
development. For example,, the paraprwfessionals need to be 
trained for the teacher leaders to lead, and resources must be 
available icr the new professional teachers to work face to face 
with each other. Structural changes often are risky "go for 
broke" enterprises which have the danger of total dismissal, even 
of promising elements. 

The career ladder, dossier-promotion elements in these 
districts represent incremental provisions which are designed to 
later be included in major structural changes for teachers. The 
innovative features are intended to be forerunners of more 
substantial change for classroom teacher working conditions. 
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SECTION THREE: IMPLEMENTATION OUTCOMES 
District Results 

Park City 

The District employed 67 teachers during the first 
implementation period. Of these, 55 were eligible for promotion 
because of years of experience . Forty-four eligible teachers 
gathered data; five non-eligible teachers gathered data. Of the 
44 eligible teachers, 40 (60% of District teachers) were promoted, 
3 were denied by Panel review (7% of those reviewed), and 1 
teacher gathered, but did not submit, materials for review. 

In the second year the District had 66 teachers. Six 
teachers submitted dossiers for promotion* Five were approved and 
one was denied. Two of those approved had been denied in the 
first round of review. At the end of the second year 75% of the 
District teachers had been promoted to the Associate Teacher 
level • 

Near the end of the second implementation year the career 
ladder committee for the District decided to keep the 
dossier-promotion system. Alternate schemes had been investigated 
throughout the year which might (a) reduce teacher anxiety and (b) 
provide greater confidence for teachers in the evidence gathered. 
In particular, the committee looked for classroom observation 
approaches. 

High priorities for future development of the 
dossier-promotion system in the District are (a) creating teacher 
leadership positions, (b) expanding the pilot systematic 
observation system , and ( c ) devising a workable merit pay 
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component to Beet State mandates. 
Webo 

The Hebo District employed 635 teachers during the 
implementation year. Of this total, 341 (54%) submitted dossiers 
for review. The regional promotion panels approved 333 candidates 
while rejecting 8 (2.3%). 

The District career ladder committee decided late in the year 
to retain the system for the future. Highest priorities for 
development included (a) refining the student achievement line of 
evidence, (b) raising the total number of Associate Teachers, (c) 
better defining the job enlargement component, and (d) devising 
teacher leader positions. 
Rich 

Rich School District, with 29 teachers, was the smallest 
district in the implementation. During the first year 19 (65%) 
teachers submitted dossiers and were promoted (none denied). Mo 
teachers elected to fill job enlargement opportunities; money set 
aside for this feature reverted to the promotion stipend. 

The career ladder committee decided to maintain the dossier 
promotion system. Work was begun on (a) adding the Teacher Leader 
position for subsequent years, (b) revision of several assessment 
forms and procedures, and (c) adding a merit component to be 
granted by the principals to 4-8 District teachers each year. 
Granite 

Of the 3,271 teachers in the District, only 550 (17%) 
submitted dossiers for review. The quota for awards was 509. 

The merit pay program in this District was competitive, 

o 18 
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quickly introduced, and not integrated with the vertical or 
horizontal features of the career ladder. Following this 
experience a task force for the District career ladder committee 
recommended changes for the next year which included dropping th* 
dossier review system in favor of a classroom observation scheme 
(AIM) • A participant observer's analysis of the program in one 
junior high school is included as Appendix E. 

Additional Analysis Related to this Implementati - 
Project deadlines did not permit a full analysis of the 
evaluation procedures or teacher satisfactions of this 
implementation. However, this analysis is underway and will be 
reported in the research literature. Additional analysis will 
include: 

data on individual lines 

lines of evidence interaction 

reliability of dossier judgments 

evaluation strategies for "special assignment" 

and minorities teachers 
teacher satisfaction 
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SECTION FOUR: CONCLUSIONS AMD RECOMMENDATIONS 

1. Teachers can control their own evaluation. They are able to 
make judgments about what is appropriate for their situation 
and what is not. However, it takes time and experience for 
teachers in a district to develop professional expectations 
about what is quality practice, and what constitutes 
sufficient evidence. 

2. Teacher evaluation is largely a sociological and political 
problem i technical data gathering aspects are an important 
part. The reaction of teachers to an evaluation system 
determines its initial success to a greater extent than does 
the technical adequacy. 

3. Good evaluation is complex and requires resources, it calls 
for time, money, and creativity. 

4. Introduction time of sophisticated evaluation is considerable. 
A good evaluation system equires experience, lore, examples, 
and tradition. Teachers begin very unfamiliar with evaluation 
ideas and standards. 

5. Inservice is in teacher evaluation is crucial. Technical 
knowledge of teachers is limited. The bulk of teacher 
opinions and decisions are based on idiosyncratic experience 
rather than knowledge of good practice, alternatives for date 
gathering, processes of decision making, and standards for 
quality. 

6. A high percentage of teachers in a district are able to 
present quality data. While style-preferences may vary, on 
the order of 70% of the teachers in the participating 
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districts are working veil with different approaches. 

7. Competition among teachers in terms of Baking a case for 
quality performance creates difficulties. While teachers in 
current practice do not cooperate a great deal, they are very 
protective in terms of perceived threats to their 
collegiality. Present interactions are not extensive, but 
they are dear to teachers. 

8. Replacement of group mass protection relationships with an 
individual case judgment system, as was done in this study, 
faces an initial resistance but has a potential for a stronger 
struture for teachers in the long run. 

9. Teacher confidence in lines of evidence takes time. Teachers 
are able to see the holes in any line, and draw quick 
conclusions about long term problems. 

10. Numerical guidelines are helpful; may mislead. Teachers, 
especially ones new to the process, are uneasy over pure 
judgmental systems. The initial comfort of "what is required" 
is difficult to replace. 

11. Where job enlargement (additional pay for additional work) and 
promotion stipends compete, teachers will select promotion 
pay. 

12. Teacher leadership remains an ill defined concept. There are 
few opportunities for teachers to lead in any systematic 
sense. 

13. Merit pay is a serious threat to development of career ladder 
systems. 

14. Involving teachers over time in system development and 
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decision Baking is important. 
15. A number of new legal questions are raised by these changes in 
teacher evaluation (see APPENDIX A). 
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Park City's "permissive" evaluation 
system encourages teachers to choose 
evidence to document their value to the 
district and make them eligible for 
promotion. 



In 1984, the Utah legislature gave | 
each school district in the state the 
go-ahead for setting up a teacher 
incentive career ladder program un- 
der state offke of education guide- 
lines. Such a program would provide 
monetary rewards to teachers for addi- 
tional duties or for excellence in class- 
room performance. The challenge fac- 
ing the Park City School District and 
their University of Utah collaborators 
was to develop a teacher evaluation 
system that was believable by teachers, 
community members, legislators, par- 
ents, and researchers in what sociolo- 
gists have described as a hostile work- 
place climate for such activity. As in 
other parts of the country, district 
teachers were responsive to the idea 
of an evaluation system for promoting 
and rewarding teachers, but they were 
skeptical that it could be done. 

A major assumption of the Park City 
system was that observation by princi- 
pals should not be overused in teacher 
evaluation because of its substantial 
limitations. The district decided to 
continue to use the valuable evidence 
currently provided by principals but 
not to stretch their role with fantasies 
about increasing their observation 
time and heightening their discrimina- 
tory powers. 

The system planners also realized 
that quality teaching can be recog- 
nized in a variety of forms, or lines, 
and is not confined to a narrow collec- 
tion of strategies or competencies. The 
lines of evidence, an emergent ap- 
proach to evaluation, document and 
acknowledge teacher performance 
from several different points of view. 





Based on the assumption that no sin* 
gle line of evidence can disclose the 
value of all teachers, the approach 
incorporates evidence from a number 
of different perspectives, such as re- 
sults of a parent survey, or pupils' 
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scores tm standardized tests (sec Fig- 
ure ) >. 

To orient teachers to rhc complex 
p.*>cedures and requirements of the 
evaluation system, the district provid- 
ed inservice courses and technical ad- 
visors. Pretests given to the Teachers 
indicated that they lacked the required 
knowledge and attitudes to participate 
successfully. Posttests following the 
teachers' orientation to the system re- 
vealed encouraging growth. 

How the System Works 

The lines of teacher evaluation evi- 
dence are crucial to the Park City 
system. A dossier, usually containing a 
minimum of four lines of evidence' 
and ranging in length from IS to 30 
pages, is kept for each teacher. This 
dossier represents the teacher's best 
evidence of his or her value to the 
sch<*)l district. 

The teachers begin by selecting sev- 
eral lines of evidence for accumula- 
tion. They develop each line over 
time, giving careful attention to the 
research literature, to their own ratio- 
nale for selecting a particular line of 
evidence, and to the ease with which 
the evaluation panel will be able to 
use the evidence to paint a coherent 
picture of their performance. Which 
evidence the teachers pursue depends 
upon its appropriateness and availabil- 
ity, but teachers understand that they 
will later need to give it to a panel 
governing promotion. 



"A dossier, usually 
containing a minimum 
of four lines of 
evidence and ranging 
in length from 15 to 30 
pages, is kept for each 
teacher/' 



Figure 1. Lines of Teacher 
Evaluation Evidence 



• Pupil report 

• Parent survey 

• Peer review of materials 

• Teacher tests 

• Systematic observation 

• Administrator report 

• Student gain data 

• Professionalism 

• Special service 

• Other 



Park Cicy assists teachers in learning 
to use district forms to gather evi- 
dence and to organize it in their dos- 
siers. For example, a teacher might 
call for a pupil report near the end of 
the year. A time is scheduled for an 
impartial data-gatherer to visit the 
teacher s classroom for five minutes, 
during which the teacher leaves the 
room. The data-gatherer uses the ap- 
propriate form for taking evidence 
from the teachers students. The re- 
sults are tallied on two copies of the 
form, one of which is given to the 
teacher to inspect. If the teacher ap- 
proves of the results, the other copy is 
placed in the teacher s promotion dos- 
sier. If not, the teacher may retain both 
copies of the form. This procedure is 
used for all lines of evidence, even 
administrator visits. 

The dossiers are evaluated by a pan- 
el on promotion, which consists of 
four teachers, two administrators, and 
two parents. The panel is not aware of 
evidence teachers have chosen not to 
include. Their overriding concern is, 
"Does the dossier present compelling 
evidence that justifies a promotion? 1 ' 
Five out of the eight panel members 
must vote favorably for the teacher to 
be promoted, and the panel may iden- 
tify and reward truly exemplary prac- 
tice as evidenced in the dossiers. Pro- 
motions are not competitive, and 
there are no quotas. 

Promotions are valid for five years, 
during which time principals conduct 
routine evaluations. At the end of this 
period, teachers must be reviewed 



again to stay in place or to advance on 
the career ladder. Promotioas dr not 
require additional duties but nuke 
teachers eligible for a small number of 
optional job enlargement opportuni- 
ties, such as writing curriculum or 
handling some piece of state Depart- 
ment of Education compliance busi- 
ness. Teachers given the opportunity 
to execute these responsibilities are 
remunerated on the basis of a con- 
tract 

The Permissive Nature 
of the System 

The Park City evaluation system Is 
"permissive" in that teachers control 
which evidence to present to make 
their best case. Observers external to 
the district have criticized this permis- 
sive approach, but teachers, familiar 
with evaluation procedures that per- 
mit choice, have adopted it. Some 
observers have also objected to teach- 
ers selecting evidence. For example, 
teachers who think parents have little 
to contribute to teacher evaluation do 
not need to use that line. Similarly, 
those who regard teacher tests as irrel- 
evant need not report scores. Such a 
nonthreatening beginning has f wen 
teachers confidence with the result 
that they are currently using more 
lines of evidence than are being used 
in any other career ladder plan. Nearly 
half of the teachers use teacher tests, 
student and parent surveys are com- 
mon, and pupil achievement data are 
included in more than half of the 
dossiers. 

The system s permissive nature has 
also increased the number of accept- 
able lines of evidence. Lines required 
of all teachers must have nearly unani- 
mous support Few lines enjoy such 
acceptance. Discriminating use of 
lines of evidence avoids logical traps; 
for example, creating difficult-to- 
defend prescriptions for all teachers 
on the basis of specific instances of 
teaching excellence. 

Finally, the Park City system encour- 
ages professional behavior. It is the 
teachers* responsibility to demon- 
strate their value, and they have be- 
come involved in each other s assess- 
ments and in discussions about what 
constitutes value. 
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Program Difficulties and 
Benefits 

The Park City design is not without 
problems. It is a relatively cumber- 
some system, especially at first, with 
many unfamiliar details. For example: 
•All lines of evidence were not fully 
developed. 

• Introduction of data gathering and 
accompanying instruction can be 
expensive. 

• Use of multiple lines of evidence 
requires teacher sophistication. 

• Necessary teacher support re- 
quires that benefits be seen, not just 
described 

• Initial use threatens teacher secur- 
ity because of independence from fa- 
miliar support of principal and col- 
leagues. 

•Teachers not electing to partici- 
pate report feeling isolated. 

• The design is not especially effec- 
tive for diagnosing problems in teach- 
ing and prescribing improvements. 

On the plus side, Park City teachers 



fbotc&tfti by Non Scboiot 

have realized nonmonetary rewards in 
addition to promotions. Many have 
reported the satisfaction of what Lottie 
(1975) called "authoritative reassur- 
ance" about their work, which comes 
from documenting their impact. The 
results of a number of the lines have 
gained publicity in the community. 
For example, NTE scores of the teach- 
ers* electing to take the exams were 
quite high, and parents have appreciat- 
ed being asked for their perspectives. 

The Parte City administrators real- 
ized that the decision to use evaluation 
to document good practice would 
have significant consequences. By 
making professional expectations ex- 
plicit, the program has been able to 
document the value, impact, and merit 
of good teaching. At the same time, it 
has relieved pressure on principals by 
easing their responsibility as sole 
judges of teacher value. Now that par- 
ents, legislators, and critics know the 
good things that are going on in the 
schools, both they and principals can 



"Now that parents, 
legislators, and critics 
know the good things 
that are going on in 
the schools, both they 
and principals can 
support teachers 
more effectively." 



support teachers more effectively. 

The Park City approach has shown 
how teacher evaluation and promo- 
tion can affect the sociology of the 
teaching workplace. Providing teach- 
ers with a shared professional hurdle 
has also heightened their sense of 
shared professional identity.D 
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APPENDIX Bs TEACHER EVALUATION AND PROMOTION HANDBOOOK 



v Introduction 

The teacher evaluation system is a Dossier-Promotion type. This 
system has a number of features: 

— Performance is documented by individual teachers in a 
dossier which they prepare and present for review. 

— The value, merit, and impact of the documented performance 
are determined oy a Panel which represents different 
District points of view, but is dominated by peer teachers. 

— Individual dossiers contain a variety of lines of 
evidence; they are not consistent among teachers. 

— A satisfactory review leads to a promotion which has a 
tit Is, some permanence (re-review required in five years), 
and no specified additional duties. Promoted teachers do 
have additional expectations for professional 
responsibility. 

While the major immediate purpose of dossier preparation is a 
review for promotion, dossier assembly and maintenance as a form of 
teacher evaluation has additional benefits for teachers and the 
educational system. An up-to-date dossier can be a source of 
security for teachers . because it is an alternative to yearly, but 
superficial, administrative review. The multiple lines of evidence 
contained in a dossier present a broader, and more customized, 
picture of teacher performance than single line of evidence systems 
such as observation checklists or clinical supervision approaches. 
Evidence systematically collected for individual dossiers can be 
compiled to give a useful view of current teacher performance in the 
District. Many of the lines, such as pupil report, are more reliable 
when they present data from many years of practice. Finally, the 
exemplary practices of individual teachers can be documented. For 
these and other reasons, teachers who are promoted will be encouraged 
to update their dossiers each year. 

The Dossier-Promotion system is based on a number of ideas of 
teacher evaluation. First, good teaching is a complicated 
phenomenon, and comes in a variety of forms. There is relatively 
little value in a priori descriptions of good teaching, but crreat 
value in recognizing quality performance when it occurs in a context 
of need. Next, there are no single foolproof methods for determining 
teacher merit, value, and impact, in addition, all lines of evidence 
and contributing observers have biases which require the balancing of 
multiple audience views. Finally, the central task in a Dossier 
evaluation system is to assemble the best pertinent objective data 
possible for each teacher under consideration. 

The system emphasizes teacher choice and control. Teachers are 
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expected to select the most pertinent lines of evidence. They 
inspect evaluation data before submitting then to the Promotion 
Panel. The promotion decision is made on the teacher's best case of 
evidence. This "permissive" approach enables a wider variety of 
evidence lines to be used in the larger District system, but does not 
diminish the challenge to the teacher to demonstrate excellent 
performance. Fairness in the Dossier-Promotion system does not mean 
that all teachers are evaluated with the same evidence; it does mean 
that all teachers have an equal opportunity to document their merit. 

The Dossier-Promotion system recognizes the subjective nature of 
teacher evaluation; it places this phase in a panel which considers 
each individual case. It should be recognized that other evaluation 
systems (such as clinical supervision, competency-based, 
administrator checklist, test-based) place their subjectivity in the 
activity of selecting initial criteria before actual performances are 
examined. 

The minimum standards suggested for most of the lines of 
evidence need to be carefully used in order not to change the main 
evaluation task: documentation of quality performance. Minimal 
standards can be misleading in the question of determination of 
quality. The focus of the evaluation system can be either (A) show 
evidence that exceeds minimum standards, or (B) demonstrate quality 
performance. The dossier system task is clearly the latter. 

The system takes its legal and professional precedents from the 
procedures commonly used in higher education. It differs 
significantly from these procedures in. that it emphasizes the 
certification nature of the promotion by including non-professionals 
on the decision making Panel. 

The following sections will give an overview of the dossier 
preparation process, give specific guidelines for each of the lines 
of evidence, describe the decision-making process, and present 
evaluation forms used in the District. 

THE PROMOTION DOSSIER 

1. Assembly of the dossier is the responsibility of the 
candidate teacher. 

2. Dossiers must be submitted two weeks prior to review. 

3. Teachers are eligible for promotion who have completed at 
least three years of successful teaching, including one year in the 
District. 

4 • Lines of evidence prepared by third parties (i.e., 
administrator report, peer review, student report, parent survey, 
teacher tests, systematic observation) will require the candidate to 
secure a copy for the dossier. 

5. Dossiers should not be longer than 15 pages. They should be 
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bound and have a durable cover. They are considered to be a 
professional representation of a teacher's work; attention should be 
paid to the appearance of the dossier* 

6. Pive copies of the dossier are required for review. Pour 
copies will be returned to the candidate. One copy will be kept in 
the District Office by the Promotion Panel in case of appeals and to 
provide information for future Promotion expectations. While 
absolute secrecy is not promised, the retained dossiers are 
considered confidential, and are for the use of the Promotion Panel 
and their staff only. 

7. The candidate is encouraged to assemble one support document 
file, of any size, which contains materials that back up dossier 
contents and provide additional information. The Promotion Panel may 
or may not use the support document file, but it is in the interest 
of the candidate to provide as much material as possible. 

SUGGESTIONS FOR ASSEMBLING A DOSSIER 

A dossier represents a teacher ■ s best case for quality 
performance. Preparation of a dossier begins with a teacher thinking 
about their tasks and results that are of value. The activity of 
assembling a dossier is a process of documenting these activities and 
outcomes. Because teaching is an open-ended and situation-specific 
job, the cases made by different teachers (and their dossiers) will 
look quite different. The various lines of evidence are less 
expectations and requirements, and more some fairly common ways in 
which teacher tasks and results can be documented. 

It is best to begin dossier assembly as early as possible, and 
to add to it as time goes on. Some clear thinking about what is 
important in ones work is more helpful in the initial dossier design 
than trying to guess what the Promotion Panel will look for or 
finding out what others have done. Successful applicants have 
reported a range of six to 100 (!) hours to be required for dossier 
assembly. The most common seems to be a total of two working days. 

Early arrangements need to be made to take the National 
Teachers 1 Exam, participate in systematic observation, or prepare for 
peer iieview of materials. Discussions with consultants, Oversight 
members, and promoted teachers all are helpful in assembling the 
dossier. Many applicants have said that their ideas for a good part 
of the dossier contents occured to them in the middle of dossier 
preparation. Finally, leave plenty of time for final dossier 
preparation (typing, signatures, early reviews;. 

The support document may contain transcripts, letters, lists and 
other materials which give credibility to evidence presented in the 
dossier. The support document does not have to be bound. 

PROMOTION PANEL 

The Promotion Panel has eight voting members for each review. 
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The members are four teachers # two administrators , and two community 
representatives. Two teacher members will be from the candidate's 
school* and one teacher from each of the two other District schools. 
Five affirmative votes for promotion are required* 

Teacher members of the Promotion Panel pool will be elected by 
secret ballot from their respective school faculties. Each school 
will have two Promotion Panel pool members: a Pull member and an 
Alternate member. Full members vote on all dossiers presented to the 
Panel. Alternate members vote only on candidates from their own 
school. Alternates may serve in an absence of the Full member. 
Teacher members are elected for two-year terms: the first year as an 
Alternate and the second year as a Full member. A new Alternate will 
be elected each year at each school. 

Administrator members of the Panel are the Superintendent and 
principal of the candidate's school. Other principals may take their 
place in an abs&nce. 

Public members are appointed by the School Board, nominations 
of public members may come from school-community councils, PTA, 
principals, or teachers. Public members serve a three year term. 

Each meeting of the Promotion Panel will have an acting Chair 
which rotates among the Full teacher members and Superintendent. The 
Chair is responsible for managing the meeting, distribution of new 
dossiers, collecting and counting ballots, and preparing notification 
letters. 

The Promotion Panel vote is by secret ballot. It has three 
possible findings: 



If promotion is denied, -the applicant remains eligible for revie%* in 
subsequent years. 

"Clarification needed" me&ns that the Panel will vote again on 
promotion, but requires additional information or clarification on 
specified lines of evidence which the candidate will be asked to 
furnish. The candidate may not add new evidence. If the total 
number of affirmative votes combined with "clarification needed" 
votes equals five or more, the applicant will be asked to provide 
additional information about the specific items under question. 

The Promotion Panel will provide a written notification of the 
decision, including comments on strengths and weaknesses, for the 
candidate on the next school day. 



The lines of evidence of teacher performance presented in this 



1. 
2. 
3. 



Promotion 
Promotion denied 
Clarification needed 



Part II: Use of Lines of Evidence 
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section were designed to be used in a dossier teacher evaluation 
system. Mo. one line is adequate to give a satisfactory picture o£ 
teacher performance, thus the use of multiple lines is recommended. 
Bach line has limitations in focus and procedure which means that 
care must be exercise! in selection and use. None of the lines is 
appropriate or fair for every teacher in a district; requiring their 
use can lead to lack of teacher support and technical difficulties. 

Teachers should consider which lines fairly and appropriately 
document their performance. The lines can be used for formative or 
summative evaluation. It is recommended that teachers inspect data 
collected by these lines, and make the decision to use them in 
summative evaluation. 

STUDENT REPORTS 

I. Why consider student reports? 

Students are accurate reporters of some, but certainly not all # 
parts of a teacher's work. Pupils are familiar with teacher 
performance, quality teaching is in their interest, they know their 
own case well, and they are quite an inexpensive source of 
information about teachers. Studies have indicated that students can 
make effective and consistent judgments. In particular, they are 
able to distinguish between teacher effectiveness and merely liking a 
teacher. 

Pupil surveys need to be carefully designed and used to justify 
them as a line of evidence about teacher quality. Item selection is 
crucial. Topics of "opportunity to learn" and "global* items work 
well. On the other hand, many topics commonly found in pupil rating 
forms should not be used, including: popularity items, style, 
teacher knowledge, and personal characteristics. A representative 
sample of student opinion is essential. 

II, Pupil Report Procedures 

Students should be surveyed near the end of classes (for year 
long courses in April or May), Surveys in the last two weeks of a 
term should be avoided. Teacher should schedule an outside survey 
administrator (UTEP staff, clerk provided by District) . Standard 
District forms will be used; items ?*re presented in Section Four. 

It is important that teachers prepare their classes for survey. 
Students need to know that the data gathering is important, that it 
makes a difference to the teacher and the system. A frank discussion 
may be helpful. Avoid pressuring the students, but let them know 
that their cooperation is needed and expected. 

III. Standards 

Pupil reports should be gathered on all elementary students and 
at least half of middle- or secondary-school candidate's classes. 
Multi-year data are preferable. 
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The minimal standard chosen to qualify as an acceptable line of 
evidence is an average rating on the global item ("this is a good 
teacher") at least one standard deviation above last years' mean for 
the appropriate grade level: 



Grade Level 1984-85 mean (SD) 

K-3 1.15 (.15) (3 item scale) 

4-6 1.32 (.22) 

middle-school 1.70 (.37) 

high-school 1.60 (.30) 

Scores higher than one standard deviation below mean will considered 
strong. 

PARENT SURVEYS 

I. Why use parent input in teacher evaluation? 

Parents can provide a part of the picture for determining 
teacher quality in many, but not all, cases. One teacher 
responsibility is to give information about classes and students. 
Parents can give useful and accurate feedback about student reactions 
to classes in areas such as expectations and challenge. Past parent 
evaluation has consisted of much hearsay and isolated cases of praise 
or criticism which are difficult to use to determine teacher quality. 
Parent surveys provide a much more systematic and representative view 
of teacher performance. 

As important as parent views are, they have limitations in 
teacher evaluation. Mere popularity with parents is positive but not 
compelling in determining teacher excellence. Items must be selected 
for parent surveys with great care. While most parents are expert in 
raising their own children, they are in most cases not expert at the 
tasks of a classroom teacher. 

II. Using parent surveys 

The District parent survey form (in Appendix) must be used. 
Teacher should ask school Oversight person for administration of the 
forms near the end of the term or course (or near Panel review time, 
if necess«ry). Forms will be distributed and returned by mail. 
Forms will be scored by outside party. Results are to be inspected 
by teacher before they are submitted to administrators or Promotion 
Panel. 

Teachers should inform students that surveys will be sent home, 
the procedure is an important one, and students can help to remind 
and encourage parent participation. 

III. Standards 
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The main criterion will be the mean score on the global item, A 
minimum average of one standard .deviation above the mean has been 
selected. An average one standard deviation below the mean will be 
considered Strong. 

Grade level Mean (SD) 

elementary 1.41 ( .28) 

middle t secondary 1.59 (.5)} 

All parents of elementary school teachers should have an 
opportunity to complete forms. Half of the classes of middle- and 
secondary-teachers will be selected to receive forms. 

Representativeness requires at least 12 parent responses from 
each of three independent classes. The current career ladder 
installation pressures mean that elementary teachers will have to be 
given some logistical consideration. Patterns of parent response 
over the past three years is the strongest evidence. 

STUDENT ACHIEVEMENT 

I. Why use student achievement? 

One of the most important indicators of teacher quality is that 
students learn. It is often said that we "don't care much about how 
a teacher works, what classes they have taken, or what organizations 
they belong to — if the kids are learning 1" However, it should be 
remembered that while good teaching does make a difference for 
students, teachers do not have direct control over student 
achievement. For example > prior achievement (pre-test scores) 
account for 60% of the variance (not amount) in st ident gains. 
Student motivation, parent support, and school facilities all play a 
role in student achievement. Thus, pupil achievement is an important 
line of evidence but a very difficult one to attribute to teacher 
performance. 

In spite of the problems with linking student gain to teacher 
activity, some useful estimates can be made for purposes of teacher 
promotion. Two routes have been developed: a "purist" technical 
approach, and a "practical" strategy which has shown promise for 
current needs. 

II. Student Achievement Data 

PLAN A; Teacher provides pre- and post-test data on: 

Part 1: a. Four major class goals for the year 

b. Two major goals from a single teaching 
unit (3-6 week duration). 

— Goals must be validated as "major" by principal, 
department Chair, or three peer teachers . 



ERLC 



APPENDIX B p. 8 



— Measures may be teacher made, departmental , or 

standardised. They may be paper-pencil, inventories, 
sample products, performances, objective-item or 
subjective-item, norm- or criterion-referenced. 

Measures must be validated as satisfactory by principal, 

department Chair, or three peer teachers. 

Part 2: A description of what gains mean: what students now are 
able to do that they could not do at the beginning. What 
significance these learnings will have in the future. 

PLAN B: Teachers present standardized test data, pre- & 
post-measures, comparisons with national norms, gains adjusted for 
prior achievement. 

III. Standards 

Students should show significant progress on the major goals. 
There should be evidence of increased competence and greater 
knowledge. The educational significance of the gains should be 
apparent. The Panel will place the results in one of four ratings? 

Strong—muiti-grade level gains, outstanding progress, 
impressive conparisons with other groups 

Satisfactory — documented gains, clear educational valje, 
important topics and goals 

Weak — gains apparent, but little comparative data; minor 
goals, short term 

No value — gains unclear; trivial or minor goals; no 
merit comparisons 

PROFESSIONALISM 
I. Why consider professionalism? 

Documentation of professional activity is a good indicator tba 
a teacher is prepared to teach well, works to keep qualit 
performance, and is up-to-date in their practice. In addition 
teachers are expected to support good practice of colleagues 
participate in the larger school program, and contribute towar 
educational concerns outside of individual classrooms. 

We say that a teacher is a good professional when they do thing 
like the following: 



they are self-cricital about their practice, they evaluate 
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beat quality curriculua and instruction, colle-9u*s are 
supported in good work, the larger school progra* is 
supported, and they take responsibility for educational 
concerns outside of their classrooa. 

A teacher does not have to do all of the above to be considered a 

9 ?° < \* 1 P e88iona1 ' but certainly to exhibit substantial and 
significant activity in this direction. 

Of course, professional activity does not tell the whole story 
of teacher quality. The main teacher activity is in the classroom. 
Mere totals of outside experience and contributions do not 
necessarily contribute to teaching effectiveness. However, evidence 
of professionalism does speak to a teacher's readiness to perform and 
their contribution as a member of the profession and school district. 
Finally, profersionalism is a relatively easy line of evidence to 
prepare and to interpret. 

II. Content of the professionalism line 

Iri order to document their professionalism, teachers should 
develop a resume which can include items like the following: 

1. list of classes taken with dates and description (transcripts 
can be included in a support document ) ; 

2. list of professional organizations and offices held in these 
organizations; 

3. list of community activities involved in and description of 
your role? r 

4. descriptions of particular ways that you hav 2 assisted 
colleagues in the performance of their duties, include names and 
dates; 

5. descriptions of special services you have performed fior the 
school or district; 

6. list of visits, consultations (include dates and names); 

7. list of any special training activities participated in; 

8. advanced degree programs. 

include activities from the beginning of the teaching career. The 
resume to be included in the dossier should not exceed two pages. 
Put support documents (transcripts, letters, etc.) in a support 
folder which the Panel may review. 

III. Standards for quality 

Teachers may show professional activity in inservice, degree 
programs, District innovations, professional organizations, District 
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service which extends beyond the local classroom and program, support 
of colleagues r and community activities. Activities in the past five 
year 8 are most important, but earlier accomplishments may be taken 
into account to make the case of long term professional quality. 

Panel will place findings in one of four categories: 

— Strong: consistent activity in more than four areas; 

updating of skills in continuous in service? 
postgraduate degrees ; leadership 

— Satisfactory: consistent activity in several areas, 
with significant involvement in others 

— Weak: uncoordinated inservice; unclear patterns of 
involvement; few areas of activity 

— No value: no areas of significant/ consistent 
involvement, 

TEACHER TESTS 

I, Why consider teacher test results? 

Teacher tests, like the National Teachers Examination, are 
designed to sample teachers 1 knowledge . An important part of teacher 
quality is that they "know their stuff," i.e., have a good grasp of 
the subject matter which they teach and be adept in basic skills such 
as reading, writing, and listening. In addition, there is concern 
that teachers know the growing body cf information about professional 
practice. Three teacher test areas are (1) subject matter, (2) 
communication skills, and (3) professional knowledge. Public and 
legislative expectations clearly are for teachers to perform well on 
teacher tests, a number of states mandate teacher test results as a 
part of certification or employment. 

A number of testing programs provide objective tests written for 
public school subject areas. Middle- and high-school tests focus on 
specialty areas, such as German, Physical Sciences, Industrial Arts 
Education, and Social Studies. Elementary school teacher tests 
reflect the general knowledge areas taught in the early and upper 
graces such as mathematics, language arts, social studies, and 
science. 

The teacher test line of evidence has, of course, a number of 
limitations for evaluation purposes. First, command of subject 
matter is no guarantee of classroom performance. There is no doubt 
that quality teaching requires a great deal more than knowledge of 
subject matter. Second, some teachers do not test accurately because 
their examination taking skills ere inadequate or have declined. In 
spite of these limitations for evaluation, the use of teacher tests 
in a district evaluation systems permits substantial recognition and 
reward for teachers who do well on these measures. 
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II. Using teacher teste 



The following tests are recognized for this line of evidence: 



NTE Test of Communication Skills 
NTE Test of General Knowledge 
NTS Test of Professional Knowledge 

NTE Specialty Area Tests (approximately 25 available) 
Graduate Record Examination Advanced Test in Education 
California Basic Educational Skills Test 



A special administration of the NTE will be arranged for the 
District. 

III. Standard? 

Minimal scores are expected to be at the 60%ile. Higher scores 
than this minimum contribute to the candidate's case of excellence. 
Test scores should be within the past five years. 



I. Why consider a peer review? 

Peer judgment about teacher performance is an essential feature 
of a district evaluation system. Teachers are in the best position 
to know expectations , resources, standards, possibilities, needs, and 
examples of good performance. While their perspective can be much 
more realistic than that of other audiences, teachers can include a 
vision of what practice ought to be. Teachers are more apt to spot 
exemplary activities or strategies in their colleagues work, because 
they are aware of the challenges and varieties in the field. 

Peer review is a difficult evaluation practice. When based upon 
class visits, it is no more reliable than the reports of others. 
When based on a reliable number of visits, it becomes prohibitively 
expensive in terms of good teacher time and money. It is subject to 
bias of friendship, politics, and style preference. Limiting peer 
review to materials has made a defensible line of evidence. 

II. Using peer review 

A teacher should collect materials for a peer review over as 
long a time period (up to two years) as possible. This is so that 
the preparation will not be too demanding at any given time. 
Materials may include: 



PEER REVIEW OP MATERIALS 



sample student products 
quizzes, exams 
lesson plans 
assignments 
grades , records 
laboratory exercises 



pre-, post-test results 
curriculum plans 
photos of room 
feedback to students 
calendars 
media lists 
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Materials for review should be submitted in not sore than two 
boxes . 

Peer review panels will be selected from teachers outside of the 
District. At least two members of each three-teacher panel will be 
at the grade level or subject area of the teacher under review* The 
panel will complete two forms: (1) Report to Promotion Panel, and (2) 
Feedback to Teacher. The Report form will have 'three categories of 
findings: 

-Evidence of a Well Functioning Teacher, 

-Well Functioning Teacher, with Exemplary Practice(s) 

-Insufficient Evidence of Well Functioning 

The Report form will be signed by all three teachers. Teacher under 
review will see Report, and decide if it is to be shown to any other 
person. Feedback to Teacher will provide specific comments and 
reactions for the teacher's use only. More details for this line of 
evidence are civen in Section Four. 

III. Standards for Review 

The minimum finding required for using this line of evidence for 
promotion will be "Evidence for Well Functioning." Findings of 
exemplary practices may be taken into account as strengthening the 
case for teacher excellence. 

OTHER EVIDENCE 

I. Why include other evidence, not dealt with in other lines? 

Quality teaching comes in a variety of forms; some performances 
that are recognized as excellent are quite situation-specific or 
unique to individual teachers. Teachers are expected to contribute 
individual features or provisions as a part of their professional 
work. The specified lines of evidence may not include every 
indicator of teacher quality. Thus, teachers are encouraged to 
consider designing and completing an additional line of evidence 
which deals with a unique contribution, student outcome, or District 
service that extends beyond usual professional expectations. 

II. Use of "Other" as a line of evidence 

The candidate should describe the nature and impact of their 
evidence. They should carefully document needs, events, outcomes, 
and other features of their work. 

III. Standards for Judgment 

While teachers are encouraged to use this line, it is one of the 
most difficult to describe and limit ahead of time. The Panel will 
consider the idea and the evidence. It will then make a subjective 
decision that what was presented in the line represents strong and 
compelling reason for promotion. If they consider that it does, it 
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will count as one of the four required lines. If they consider that 
the evidence is without merit, this line will not count toward 
promotion. If they find that the evidence is positive but weak, 
promotion will require other four satisfactory lines, or three other 
lines — one of which is above minimal. 

Part III: Guidelines for Promotion Panel Decisions 

The following guidelines will be used by Promotion Panel members 
in their review of dossiers and decisions about promotion. 

1. There are no quotas for promotion. Numbers of 
successful candidates should not be taken into account in 
deciding about current applicants. 

2. Only information contained within dossier will be 
considered. other evidence, information, and hearsay 
about the candidate should not be discussed by, or 
influence the judgment of, individual members. The 
possible bias of individual members by external 
information is compensated for by the numbers of panel 
members and the?* various roles in the educational 
community. 

3. The judgment for promotion is based on a dossier. It 
is not on a teacher as a professional, classroom merit, or 
reputation. 

4. Deciding for promotion means that a teacher has 
presented sufficient evidence or being a well functioning, 
contributing member of the District. The evidence should 
be compelling and sufficient so as not to raise doubts in 
the Panel member's mind. 

5. Quality evidence can pertain to a teacher's (A) 
preparation and potential, (B) process and performance, 
and (C) student out cone. 

6. Lack of evidence in an area, e.g., pupil report or 
parent survey, should not be considered negative by Panel 
members. They should only consider evidence presented. 

7. Panel members who are familiar with conditions and 
expectations at specific schools can share this 
information to provide a perspective. 

8. More recent data, within the past three years, will be 
most important in decision making. Older data are less 
compelling, but valuable because they point to patterns of 
performance and accumulated merit. 

9. Judgments should not be based on comparisons of 
specific teachers with each other. For example, the 
status of teacher Jones should not influence a decision 
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about teacher Smithe. 

Understanding about the merits ©f a specific teacher may 

result from comparisons with the practice of other 

teachers in general. For example, levels of parent 

satisfaction of teacher Jones may be judged in comparison 

with averages of District teachers last year. 

10. Panel members should share their differing 
perspectives, questions, and standards which come from 
their individual experience as well as representatives of 
roles within the educational community. 

11. A Panel member should not make a decision about a 
dossier unless they are familiar with its contents. 

12. Panel members will vote by secret ballot. A record 
of the numbers of votes will be kept by the acting Chair. 

13. Each panel member is responsible not to vote on a 
dossier where there is a substantial conflict of interest. 

14. Judgments about the merit and value of lines of 
evidence data are of two kinds. 

A. Lines which have an objective or mandatory 
outcome include pupil report, parent survey, teacher test, 
peer review of materials, and systematic observation. 
Each of these has a minimal cutoff above which the Panel 
must find the evidence supportive of promotion. As 
examples, an NTE score of 73%ile or student report global 
item average of 1.3 both automatically recommend for 
promotion. 

B. Lines which require a panel judgment include 
prof essionalism f administrator report, student 
achievement, and other evidence. In these lines Panel 
members may find the evidence to suggest: 

-strong merit, value & impact, 
-satisfactory merit, value & impact, 
-weak merit, value & impact, 
-no merit, value & impact. 

15. Only "strong..." or "satisfactory..." findings will 
count toward the required four lines for promotion. 
HOWEVER, the Panel as a whole has the descretion to 
balance a finding of "weak..." with "strong" performances 
in other lines (considerably above minimal) or compelling 
lines of evidence beyond the required four. 

APPEALS 

An appeal of a promotion denial may consist of four steps: 
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1. Confer with Oversight Committee member at local school for 
clarification of process and basis for appeal. (Unsuccessful 
applicants are eligible for review in subsequent years). 

2. Meet with entire Oversight Committee to present case. New 
materials may not be added to dossier. Members will vote on whether 
or not there is a reasonable basis for appeal. Two votes (secret 
ballot) are required to have Promotion Panel reconsider the case. 
Oversight Committee prepares a written statement of the basis for 
re-review. 

3. Re-review by Promotion Panel. Applicant may present case in 
person. 

4. Further appeals are as per District grievance procedures. 

The applicant or a representative may attend any stage of the appeal. 

HOW NUMERICAL STANDARDS *ERE DETERMINED 

Numerical standards were developed for several lines cf evidence 
for 1985-86. Pupil reports and parent surveys had minimal acceptable 
levels established by using the same rules which follow: 

1. Tabulate 1984-85 data for Park City teachers. 

2. Determine averages (means) on global items for teachers 
at each of four grade levels: K-3, 4-6, middle-school, and high 
school. 

3. Test for statistical significance of differences in means 
by grade level. Grade levels which differ are to receive their 
distinctive minimal acceptable levels. 

4. Compute means and standard deviations (average distance 
from the mean) for each distinctive grade level. 

5. Rule for minimal level: a class average more than one 
standard deviation below the mean for that distinctive grade level 
group is considered to be not acceptable. 
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SUGGESTED TIMETABLE FOR DOSSIER ASSEMBLY 



Sep 



Oct 



Nov 



Dec 



Jan 



pretest 
validate 



profess- 
ionalism 

register 
NTE 



NTE 



peer review 
materials 



prepare peer 
review 



goal 8 
plan peer 
review 



Feb 



Mar 



Apr 



May 



administrator 

review 
peer 

reviews 



update 
profess- 
ionalism 



register 
NTE 



parent 

surveys 
student 

reports 
NTE 

post-test 



submit dossier 



Teachers who have already been promoted may complete parent and 
student surveys in May, update professionalism at year end. 
Administrator reviews can be scheduled later in the year. The 
District may schedule its own administration of the National 
Teachers 9 Exam. 
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granite SCHOOL DISTRICT career ladder plan 



Extended Year 
Performance Bonus 
Job Enlargement 



(2,549,346 SOX 
$ 509,869 101 
$2,039,477 401 




1. Extended Tear: Six and one-half (6-1/2) days total. One day prior to school; one day at the conluslons of 
three school quarters. These are teacher work days at the school. T* days have been returned to the students' 
Instructional calendar. One-half day used for Inservlce training to be determined by the Individual school. 
Z. Jjrfonance Bonus: Applicants submit a dossier containing lines of evidence to confirm outstanding performance. 
Minimum stipend Is $1,000.00. 
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1. Extended Tear: Seven days added. Four used for Inservlce training in essential elements of instruction and the 
regaining three days for teacher preparation. 

2. Perforwnce Bonus: Awarded to Associate and Senior educators based on acceptable dossier. 

3. Job Enlargement: Optional, by application on all levels, as facilitators, promotion review board, and building 
level positions. 

4. Teacher Shortage: Still recruiting for coming year. They plan to review the staffing after school has started. 
Anticipated use of funds will be for extension of contract to teachers and additional duties. 
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This plan includes i high level of professional Judgement In the evaluation of teacher dossiers containing a ilnlii 
of four and a maximum of eight lines of evidence. Administrator evaluation of teacher performance is the only 
clearly required line of evidence. This plan is heavily weighted toward performance bonuses (70S) and these bonuses 
are directly linked to promotion to the Associate Teacher level for as many as 821 of the district's teachers. 
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This plan Is i revision of the 1984-85 plan for this school district and, as such, contains a "change over" phase for 
1985-86. Thus, waivers will be given to most teachers in order to make the transition to the new plan. Kith little 
teacher turnover, almost all teachers will be eligible for all aspects of this plan during 1985-86. - 
1. Extended Year; Two optional days available for all teachers utilizing 101 of the available funds. 
I Performance Bonus: Associate and Senior Teachers are eligible to share 801 of the available funds. 
3. Job Enlargement: All teachers are eligible for seven identified extra assignments that will utilize 101 of 
the available funds. tr i 
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More than 800 teachers in Granite School 
Datnct will put together brag books in their 
spare time over the next several months, 

If 1 teacher's brag book includes impres- 
sive souvenirs of his or her professionalism 
Jj tocher wfll get a |l,00o bonus from the 
wool district, compliments of that now fa- 

miliar but still confusing term "tocher ca- 
reer ladders," 



to reward Granite teachel 




porter. Teachers there came up with a suu^ 
of four items they favored about the dosaer- 
based bouus and 16 items they disdained. 
The upshot of the survey was that teachers 

2 to !»« vll J«Wforwbattheydofor 
children u the classroom, not for what kind 
of a dossier they can put together, 

Michael GarbetUho oversees every ca- 

-"-^MvuMupcamngmeftra gMs JJJM* Win in Utah from the State 

H^f^'eSchoolDistrictprefers lion that goes on in the classroom between 2! S^R**"* ** 
tocallthemdosjiert tepagesofascrapbook. P^n aout of step with others throughout th* 

N book or dossier, the game is the ^ if doctors had to Wtle up all the uTl^lTt^ 
samt Teachers must spend 20 or 30 hours fencer cells they had kJUed, if pUob bad to JFj&tt 

f&lmto^miEL ^"acddMtstheyVavolded S+fc&frf" 1 ** 
strict they are a worthy investment. psychiatrists had to Une up all the people ^^nGraiu'te.hesa.d. 
Tn »i» ,„ , they had saved from suicide Other districts are doling out an average 
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wontreceiveaneitranickeL not participating in the venture numoerof teachen whogetowuses, hesaid. 

to some ways the r» is in. Before a single . dossiers are a standing joke In many ^TJ^f^^^.^ 
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Question 153; Do v nn f av0 r or odmsp th a *a c l 

1 oppose the idea nf ^ e state career ladder nrnpr.p,? 
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STATE CAREER LADDER 

DO YOU FAVOR OR OPPOSE THIS PROGRAM? 




SMWT OPPOSE 



SIR OPPOSE 



* state career Udder prog™ tor teachers is fa.ored by the aajority (65*) of 
Park City residents, hove.er one of ft,. (22*, sa, the, "don't Know". According to 
Dan Jones ■ Associates, although TO re people are knowledgeable today, the perception 
is that there is no consistency throughout the state concerning career ladders. 

DEMOGRAPHIC PROFILE: 

followI„g S subgroups: fa, ° r ° £ a » at * «"«• ^dder program tend to fall into the 

- under age 34 (1.56) 

- some coll/tech (1 .66) 

- residents who rent (1.68) 

- $25-35,000 income (1.61) 

- less than 3 years in Park City area (1 61) 

- parents with children in elementary school (1 73) 
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Question IS*: Befor e they are hired by the school district, do you feel all teachers 
should o r should not be required to pass a basic competency test to measure such 
thinRs as their general knowledge and ability to think? 



I 

ta 
o 

K 

u 




COMPETENCY TEST S_ 
SHOULD TEACHERS BE REQUIRED TO PASS? 




DEFINITELY 



PROBABLY 



' 4 a 

®9f$8 RSS ^a k>»^» 



PROB NOT DEP NOT 



DE 



The Park City general public is overwhelming in favor (86%) of a required basic 
competency test for teachers before they are hired by the school district. 



DEMOGRAPHIC PROFILE: 

Residents most adamantly in favor of a required competency test for teachers 
are inclined to be: (lower mean score indicates more in favor) 

- age 25-34 (1.31) 

- less education, more in favor (high school grads - 1.24) 

- females employed part time (1.28) 

- $25-34,000 income (1.28) 

- other Park City area residents (1.27) 

- 1-3 years residency (1.21) 

- parents with elementary age children (1.30) 
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APPENDIX Pa 

PARTICIPANT OBSERVER CRITIQUE OP GRANITE DISTRICT APPLICATION 

Jean Kron, Teacher, Granite School District 

The Performance Bonus Program (PBP), as well outlined in the 
Handbook, appears to have the potential to objectively evaluate 
teacher 8 and to reward them for documented excellence. However, it 
was negatively received by teachers; only three teachers out of the 
45 in my school participated. The purpose of this paper is to 
explore why most teachers chose not to take advantage of the 
opportunity to be financially rewarded for their excellence. Many 
teachers maintained that the PBP as conducted was not a bonus at all, 
but extra pay for extra work. 

Following the presentation of the PBP to the faculty at our 
Junior High School, our support group met to discuss concerns and 
questions about not only the PBP, but teacher merit in general. My 
report is based on six meetings held in the Fall of 1985 which were 
attended by 10 to 12 teachers each session. It also is based on 
questionnaires from the teachers' organization and District. The 
school faculty is diverse, and well represents different kinds of 
teachers (ages, backgrounds, sexes) in the District. 

From discussions and these sources, several predominant concerns 
arose based on teacher interpretation of the PBP: 

There are many teachers concerned about the [PBP's] 
delir nation of "merit pay." Compiling a credible dossier 
does not equal merit pay, which teachers define as a 
reward for excellent classroom teaching; compiling a 
dossier does equal extra work, consuming time better 
spent in the activities of classroom interaction (i.e., 
the teaching-learning process). — Taken from a followup 
letter sent to the Boards of Education and several 
legislators. 

An examination of teachers' organization questionnaires 
suggess that teachers were more concerned about the time spent 
compiling a dossier than any other factor: "with many secondary 
classloade above the 200 mark, this type of time-consuming 'busy 
work' will only add to alxeady overwhelming burdens." Or, "SOME 
REWARD! If you're really lucky, you win the opportunity for more 
work!" One teacher mentioned that "it only adds additional 
pressures within an already busy school year." 

Many teachers feel that these additional time ^.7,.- .ids, added 
to an already full day (in the classroom and at home preparing and 
correcting), will ^tract from the quality of teaching by cutting 
into those activiti. directly related to classroom interaction. 

The peer and-or administrator evaluation required by 
the ... Program does not take into account the fact that 
there is littler if any, peer-administrator evalation due 
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to the severe time restraints of the teaching day. 
Besides ignoring these very real time restraints, 
the... Program srequres more time to be expended for peer 
tabulation of surveys to document further lines of 
evidence for a colleague's dossier— in addition to the 
time spent collating one's own. 

The presentation of the PBP itself suffered a time 
constraint. During a cursory 15 minute presentation by our 
principal in faculty meeting, teachers were instructed to "read it 
over and if [we were] smart [we would] take advantage of this 
opportunity to make $1000. . .maybe. ■ Merit pay per se was not 
discussed, nor was the purpose of a dossier as a means of teacher 
mobility, independence, and documentation of excellence. Not a 
syllable alluded to the fact that a truly reliable dossier could be 
considered valid only after at least one entire year, preferably 
more, of documentation. This information should not only be 
presented in the PBP, but supported by the District. 

The concept of documentation through four lines of evidence 
has been so compromised due to financial, time, and political 
factors that several of the lines could, according to teachers, be 
considered invalid. The administrative roporfs at our school were 
filled out following a classroom visit of less than 15 minutes. 
Teachers circulated their own parent and student surveys, as well 
as tabulated their own results. The District has not supported 
evaluation procedures through dispersal and analysis of results. 
Teachers do not appear to be aware of the proper procedures, and 
if they are they apparently do not believe they will be held 
accountable fox reliable results. 

Teachers felt duped again by the District: "Districts should 
not be allowed to 'cop out' on their evaluation responsibilities 
in this manner. As usual, it gets passed down to the teachers." 
Teachers are skeptical of District programs that purport to be to 
their advantage. One teacher feels that "this prelaw appears to 
pass the buck to teachers to avoid losing career ladder monies." 

The possibility of a financial reward was the only motivation 
provided for teachers to document their excellence. Again, 
teachers responded indignantly: "Another riduculous attempt to 
make the public think that excellence in teaching is being 
financially compensated;" or, "a BONDS is not a BONUS if it has to 
be -re-earned- after the factl" or, "I can make more money with 
less effort doing other types of part time work." 

Tenured teachers felt threatened that, after 10 or 15 years 
of what they felt to be good teaching, they were asked to validate 
their competence. One teacher asked, "If I haven't been doing a 
good job so far, why have they kept me around this long?" One 
person felt that the Program presents the subtle implication that 
those teachers who do not apply lack ambition or pride in their 
profession, when in fact they do. 
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Teachers are uncomfortable in parading their classroom 
laurels and yet they dislike being lumped together as a group. 
"It's demeaning to have to pat yourself on the back." Due to what 
teachers define as poor' preparation by the District and 'a cursory 
presentation by the principal, the validity of the Program was 
constantly in question. "More effort in the preparation of a 
dossier by a mediocre teacher could make him or her appear to be 
something he or she is not. By the same standard, a poor teacher 
could eliborately fulfill this kind of program and appear 
competent . " 

As the Program now stands, teacher criticisms are 
understandable and appropriate. Teachers could not possibly have 
verified excellence with a completion deadline of only six months. 
From the dossiers I reviewed, the lines of evidence in our school 
were not properly followed. The dossier itself was an easily 
maniplated means to a financial reward without checks or controls 
for valid evaluation. 

All support group teachers did feel that merit pay is a good 
concept and should be rewarded and that a choice of multiple lines 
of evidence is a valid idea. They provided the following ideas 
for improving the process: 

1. 'Proving' of teachers should take place during their 
educational process, with certification, and before they 
are offered tenure. Not to do so is evidence of poor 
management. 

2. Formulate committees and train them in evaluation 
processes with reasonable criteria for teacher 
evaluation. 

3. More than $1000. should be considered for such awards. 

4. There should be additional steps added to the pay scale 
which, though incremental, could be awarded upon 
recommendation after a proper evaluative process. 

5. Have anonymous peer evaluation. 

6. Offer any teacher with over 14 years of experience an 
opportunity for a reasonable "bonus" evaluation. 

After reading dossiers from my school, talking and listening to 
teachers, and taking an evaluation course, I found that teachers 
as a group are uneducated when it comes to proper evaluation 
procedures and their outcomes. They are basically unable to see 
evaluation as on ongoing and vital to mobility within the 
profession as well as away from it. To remedy this, the 
presentation of evaluation programs and procedures should be 
thorough and conducted by evaluation experts. Districts should 
accept the responsibility of providing financial and logistical 
backing to train and hire experts, formulate competent peer review 
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committees, conduct evaluation workshops for teachers, and regard 
teaching professionals for valid evidence of excellence. 
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APPENDIX 6 

LEGAL QUESTIONS POSED BY CAREER LADDER EVALUATION 

Career Ladders complicate the evaluation process. Evaluation 
is increasingly not only for retention purposes, but for 
recognition of excellence. This change raises a number of 
professional questions and legal concerns which have yet to be 
answered. 

Regular teacher evaluation begins with the presumption of 
competence bestowed by a certificate. Thus, evaluation for 
retention must show evidence either of incompetence or practice 
which interferes with the educational process. There is 
considerable experience and precedent for both of these 
difficulties. However, career ladder promotion has no presumption 
of meritorious performance. it is not clear what burdens of 
documentation, opportunity, and comparison will be required for 
this new type of evaluation. 

Administrator reports clearly have legal precedents for 
teacher evaluation. However, this evaluation approach is 
discounted in the literature. It is to be expected that 
litigation will challenge principal ratings ar the only way to 
assess teacher performance. The suitability of panels to make 
decisions, including peers and parents as well as administrators, 
may be questioned. 

New lines of evidence (such as teacher tests, parent surveys, 
systematic observation) are being established. In addition to 
uncertainty about the criteria performance levels in each of these 
procedures, the use of multiple lines of evidence will raise 
questions of consistency and fairness. The novel point of 
fairness in the Dossier-Promotion system is that teachers should 
have equal opportunity to document their particular performance, 
not necessarily that all teachers be documented in the same way. 

A major cause of legal problems arising from teacher 
evaluation has been difficulties between teachers and 
administrators because of conflicting roles of administrators: 
are they leaders of the educational community or summative judges? 
Already career ladders have led to increased problems between 
teachers and administrators which can be expected to eventually 
lead to litigation. 

A final development brought on by career ladders is the need 
for much increased teacher initiative in professional evaluation. 
This change, which includes peer review, will raise new questions 
about fairness and responsibility for teacher evaluation. 
Clearly, new efforts to prepare for evaluation and to train 
evaluators will have to be made by school districts. 

An interesting dilemma for educators is the extent to which 
teacher evaluation will develop through academic procedures, i.e., 
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research and development, and how much of it will.be shaped 
through legal procedures, i.e., statute and litigation. 
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APPENDIX H: 
MICROCOMPUTERS IN TEACHER EVALUATION 
Introduction 

The new evaluation procedures will be more complicated 
for teacher 8. Not only does good evaluation use multiple 
data sources, it uses them in various combinations for each 
teacher. This is necessary because good teaching comes in a 
variety of approaches and styles , individual classroom 
settings differ greatly, and no one line of evidence is 
completely satisfactory for every teacher • Consequently , 
multiple forms and procedures must be available. Another 
complexity is that teachers will need to analyze their own 
data with statistics, graphics, and norms from other 
teachers. Finally, good evaluation will require improved 
record keeping. 

In addition to helping teachers to carry out improved 
evaluation, microcomputers can assist them to learn the new 
ideas and procedures. Inservice education in evaluation can 
be accomplished with examples, instruction, and support 
network information distributed on computer disks. 

Figure 1 presents a list of teacher evaluation tasks 
which can be assisted by a computer. 



FIG. 1: Uses of a Microcomputer for Teacher Evaluation 

Directions for evaluation procedures 

(e.g., "how to do a peer review") 
District evaluation forms, surveys 
District form letters 
Statistical analysis routines 

(e.g., means & standard deviations) 
Network data: teachers who wish to share information 
District norms on evaluation forms and surveys 
Individual teacher records 

(e.g., resume, course lists, test scores) 
Student achievement data 
Bibliographies of evaluation techniques 
Records of evaluation costs and time 
Graphics for presenting evaluation data 

Bow Computers are uped Ln Teacher Evaluation 

Microcomputers can help teachers with many evaluation 
tasks. Some of the procedures, such as gathering rating 
forms from students or parents, will be done in the same way 
by all teachers who use them. Other tasks, such as record 
keeping , will be unique to individual teachers. Teachers 
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will be expected to analyze and present their data in 
individualized fashion. 

One computer resource for teachers is a collection of 
directions for various evaluation procedures recorded on a 
disk. For example , the steps and procedures for a peer 
review of materials call for a great deal of preparation by a 
teacher. These directions and suggestions can be put on a 
reference disk. Sample items from teacher tests is another 
example of an evaluation resource. 

Specific forms used in a district can be kept on a disk. 
These forms may include student reports, parent surveys, peer 
review reports, administrator reports, teacher test 
summaries, systematic observation, and documentation of 
professional activities. While no one teacher will use all 
possible forms, having them available helps teachers to 
choose appropriate ones and encourages their use. The 
advantage of having these on a computer wordprocessing system 
is the ease with which a user can add, delete, or change 
specific items to make the forms fit his or her situation. 
Other district resources include forms for presenting results 
and decisions of evaluation persons or panels. 

Once forms are collected, they may be scored by 
computers. Card readers and sheet scanners enable forms to 
be scored on a large number of teachers and items. Most of 
these scoring programs enable aggregated data analysis which 
can be helpful in compiling district results. 

Teachers may use a number of statistical analysis 
programs written in BASIC such as computation of averages, 
standard deviations, and distributions. Still more 
sophisticated programs may be used, for example analysis of 
variance for testing statistical significance of differences 
in class pre- and post-test scores. Other procedures such as 
correlations and chi-square enable teachers to explore 
relationships among their data. Regression analyses enable 
educators to predict expected student gains, given pretest 
scores of actual classes. All of these tools allow teachers 
to better understand and use their own evaluation data. The 
advantage of the computer is to make the tools easier and 
faster to use. 



Networking information on database systems make it 
possible for teachers to contact one another. For example, 
teachers who use parent surveys can enter their own names and 
schools for consultations by other teachers who are 
considering the procedure. Resulting phone calls and visits 
of teachers by each other enable better selection and use of 
evaluation procedures. The disks can be updated by the 
district or teachers' organization. 

District statistics, such as ranges and averages of 
various surveys, can be helpful to teachers because they 
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communicate expectations and results of typical practice. 
For example, many teachers have found that records of how 
others allocate instructional time are very helpful as they 
plan for their own classrooms, Knowledge of levels of 
student satisfaction and achievement test gains shown by 
colleagues can help teachers gauge their own effectiveness. 
Availability of a wide range of norms permits formative 
evaluation by individual teachers who can compare their 
practice and results with those of colleagues. 

Custom programs 

Using a computer for evaluation goes beyond providing 
standard forms r procedures, and analyses. Teachers can 
develop their own custom records and data presentations of 
their preparation , teaching performance , and student 
outcomes. Good evaluation challenges individual teachers to 
conceptualize and document their value and impact to the 
educational system. As with other uses of computers, we 
should expect to see many examples of creativity in 
individual teacher evaluation. 
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] CATALOG 

028X VOLUME 254 

*A 002 HELLO 
*A 003 AVERAGE ft 8.D. 
*A 006 GAINS SIGNIFICANCE 
•A 008 COMPARE 2 GItOOPS 
•A 008 COMPARE >2 GROUPS 
•A 005 CORRELATION 
•A 006 DISTRIBUTION 
*A 008 GROUP CP MEAN 
•A 005 ITEM ANALYSIS 
*A 003 RANDOM WUMBERS 
•A 005 TEST RELIABILITY 
•A 004 PREDICTED GAIHS 



10 BONE 

30 335 * AVERAGE * STANDARD DEVIATION 

40 PRIST 

50 PRINT • NUMBER OF ITEMS 

60 INPUT N ' 

70 PRINT 

80 FOR I « 1 TO N 

90 PRINT *ITEM 

100 INPUT D 9 ' 

110 P m p + D 

120 M * M + D A 2 

130 NEXT I 

140 R m p / N 

Hi 5S • MEAN « . ? K 

1*0 END S -°- " ' S <* < V > 



10 
20 
30 
40 
50 
60 
70 
80 
90 
100 
110 
120 
.130 
140 
150 
160 
170 
180 
190 
200 
210 
220 
230 
240 
250 
260 
270 
280 
290 



'HOW MANY NUMBERS DO YOU WANT 



HOME 
PRINT 
PRINT 
PRINT 
PRINT 
PRINT 
PRINT _ 
INPUT HI 
PRINT 

"£™ \ LOWEST NUMBER 
INPUT N2 

PRINT •BIGHBST NUMBER 

INPUT H3 

BOMB 

FOR I • 1 TO Nl 
* - PEEK (78) + 256 * 



RANDOM NUMBERS 
(LIMIT: 1-999 INCLUSIVE) 



IHT ( RND (N) * 1000) 



PEEK (79) 



M2 THEN 200 



IF X > 
GOTO 160 

1* x < m N3 THEN 220 
GOTO 160 
PRINT X 
NEXT I 

PRINT *WANT MORE 
INPUT R 

IF R m o THEN 290 
BOME 
GOTO 70 
END 



? (YES-1 NO»0) •« 
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1 A 




3ft 


FRIST 




nil mm 


2ft 


PFIRT 


en 


print 


cn 


inn « 


TV n 
7Q 


rain 


• U 


MTM Oi 




PRIST 


1 nn 
xuu 


PRIST 


1 1 n 
XXO 


FRIST 


X20 


1SFUT i 


1 in 
X J U 


PRINT 


lift 

1*0 


PRIST 


i cn 


FRXST 


1*0 


PRIST ' 


l *>n 


FRIST 


l on 
100 


PRIRT 


i on 
190 


FRIST 


o n n 


do run 
PRINT 


210 


FOR I 1 


220 


PRINT 1 


*> o n 
230 


INPUT ^ 


240 


«r _ «r » 
J * J ♦ 


290 


V — W j. 

K * K ▼ 


o< n 


L * L + 


9 *?n 
2 #U 


M * K ♦ 


*> o n 
280 


R * R ♦ 


*> o n 
290 


NEXT I 


inn 


R2 * (N 


J xu 


PRINT 


i 

J 2 u 


PRINT 


1 1 n 


PRINT 


iin 
j * u 


PRINT 


i cn 


PRINT 1 


icn 


DBT1M 

PRINT 


i*7n 


PRINT 


i <i n 


PRINT 


ion 
J 3 0 


PRINT 1 


inn 


PRINT 


410 


END 
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coMMLktim coefficient 



TOO IIIXX SB CALCULATING TEK 



OP VALUES— OR VALUE FOR EACH VARIABLE. 



VALUB1 r VALUE2 (E.G. , 21,92) 



2 
2 
Y 



K) / SQR ((H • L - JA2) MN • M - K A2)) 



■?R2 



10 

20 

30 

40 

50 

60 

70 

80 

90 

100 

110 

120 

130 

140 

150 

160 

170 

180 

190 

200 

210 

220 

225 

230 

231 

240 

250 

260 

270 

280 

310 

320 



COMPUTING EXPECTED GAINS 



HOME 
PRINT a 
PRINT 
PRINT 

£™ I* 00 WILL 1,820 *° OBTAIN INFORMATION 
PRINT •FROM THE TEST PUBLISHER ON THE 
PRINT " PERFORMANCE OF THE GROUP ON WHICH 
PRINT "THE TEST WAS NORMED. 
PRINT "ASR FOR THE 'REGRESSION EQUATION. • 
PRINT •WHICH HAS THE PORMs 

PRINT ■EXPECTED GAIN ( Y ) *A+B* PRETEST MEAN (X), 

PRINT "THEN ENTER THE DATA BELOW I 

PRINT 

PRINT 

PRINT ■ 'A 1 VALUE" ; 
INPUT A 

PRINT ■ •B' VALUE" ; 

INPUT B 

PRINT 

PRINT "ENTER THE PRETEST SCORE OR MEAN" i 
INPUT X ; 
PRINT 
Y » A ♦ B • X 
PRINT "THE GAIN EXPECTED FOR YOUR 
PRINT "AVERAGE SCORE OR GROUP 
PRINT "WAS ";Yj"." 
PRINT 

IF X < Y THEN 310 

PRINT "YOUR SCORE WAS ABCVE EXPECTATION. 
GOTO 320 

PRINT "YOUR SCORE WAS BELOW EXPECTATION. 
END 
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20 ftUt • SIGNIFICANCE OF GAMS' 

30 rust 

40 F1OT 

50 K(X) - 0 
00 Q(X) - 0 

70 run •■umber or STUDENTS* J 
oo imot jt(x) 
f o ■ - 1(1) 
100 DDI Pi*, 2) 

no run 
120 run 

130 run •for bach studkmt enter x 

140 run •PRETEST SCORE, POSTTEST SCORE 

145 run • (E.G. , 78,04) 

150 run "NOTICE I VO EMPTY SPACES AMD 

160 Fun *A COMMA SEPARATES SCORES 

170 run 
i«o run 

190 FOR Dl - 1 TO R(I) 

200 PUn • STUDENT *;D1; 

210 IMPOT X,Y 

220 Tl - X - Y 

230 1(1) • K(I) * Tl 

240 Q(I) « Q(I) ♦ Tl a 2 

250 NEXT Dl 

*60 B - K(I) / R(I) 

270 F - R(I) - 1 

280 T - B / SQR ((Q(I) - ((K(I) A 2) / R(I))) / (R(I) * (R(I) - 1))) 
290 PRINT 

300 PUOT • ?«*?T 

310 PRINT • DF«*;F 

320 X - 1 

330 Y - 1 

340 T ■ T A 2 

350 IF T < 1 THEN 400 

360 S « Y 

370 El - F 

380 Z - T 

390 GOTO 430 

400 S - F 

410 Bi - y 

420 Z - 1 / T 
430 J - 2 / 9 / S 
440 Fl - 2 / 9 / El 

450 L - ABS ((1 - Fl) * Z a (1 / 3) - 1 + J) / SQR (Fl * Z A (2 / 31 ♦ J) 
460 IF El < 4 THEN 500 

470 X - .25 / (1 + L * (.196854 + L • (.Ubl94 + L • (.000344 ♦ L * .019527 
400 X * INT (X * 10000 ♦ .5) / 10000 
490 GOTO 520 

500 h « L * (1 * .08 * Tfc A 4 / Kl A 3) 

510 GOTO 470 

520 IF t > - 1 THEN 540 

530 X - 1 - X 

540 PRINT • P«* f X 
550 PRINT 

560 PRINT •THIS DIFFERENCE BETWEEN PRETEST AND 
570 PUNT •POSTTEST SCORES* 
5S0 IF X < .05 THEN 630 

610 PUn *IS NOT STATISTICALLY SIGNIFICANT. - 
620 GOTO 640 

630 PRINT *IS STATISTICALLY SIGNIFICANT. 
640 END 
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10 BOMS 

20 print • siG»r?:rcA*£s or difference 

22 PRINT • Z&TW&SZ TWO GROUPS 

30 nxwt 

40 nUV7 'ENTER NUMBER IAP.3BST GROUP* j 

50 INPUT B 

(0 PRINT 

70 PRIST 

80 DIM P(M,2) 

90 DIM V(2),R(2),H(2),Dt?) 

170 H - 2 

200 FOR I ■ 1 TO SGH (8 - 1) + 1 
210 V(I) - 0 
220 D(I) - 0 

230 PRIRT *GROUP*|I**:* 

240 P^IRT • NUMBER OF CASES* ; 

250 .'0T R(I) 

260 rem J » 1 TO R(I) 

270 PRIRT * CASS* ; J j 

280 INPUT P(J,I) 

290 V(I) - V(I) + P(J,I) 

300 D(I) - D(I) + P(J,I) A 2 

310 NEXT J 

320 M(I) - V(I) / R(I) 

330 V(I) - (D(I) - V(I) A 2 / R(I ) ) / (R(I) - 1) 
340 NEXT I 
350 PRINT 

430 A « (M(l) - M(2)) / SQR (1 / R(l) + l / r( 2 )) 
440 B - R(l) ♦ R(2) - 2 

450 A - A / SQR (((R(l) - 1) * V(l) + (R(2) - 1) * V(2)) / B) 
520 T - ABS (A) 
530 F - B 

700 PRINT * T-*;T 

710 PRINT * DF»*;P 

720 X « 1 

730 Y - 1 

740 T - T A 2 

750 IP T < 1 THEN 800 

760 S ■ Y 

770 El - F 

7B0 Z « T 

7^0 GOTO 830 

Z00 S - F 

810 El - Y 

820 Z - 1 / T 

8S0 J - 2 / 9 / S 

840 Fl - 2 / 9 / El 

850 L - ABS (fl - Fl) • I A (1 / 3) - 1 + J) / SQR (PI • Za (2 / 3) ♦ J) 
860 IF El < 4 THEN $00 

870 X - .25 / (1 + L * (.196854 + L * (,115194 + L * (.000344 4 L * .019527) 
880 X - INT (X * 10000 + .5) / 10000 
890 GOTO 920 

900 L - L * (1 + .08 • L A 4 / El A 3) 

910 GOTO 870 

920 IF T > « 1 THEN 940 

930 X - 1 - X 

940 PRINT * P«*?X 
95Q PRINT 
960 PRINT 

970 PRINT *TBIS . PFERBNCE BETWEEN GROUPS 
980 IF X < .05 ln£N 1010 

990 PRINT *IS NOT STATISTICALLY SIGNIFICANT. 
1000 GOTO 1020 

1010 PRINT *IS STATISTICALLY SIGNIFICANT. 
1020 END 
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SOME 



APPENDIX H 



p. a 



20 FIINT "SIGNIFICANCE OP DIFFERENCES IN AVERAGES 
30 PRINT " AMONG MORS THAN TWO GROUPS 
40 PRUT 
50 PRIM 
(0 PRIST 

70 PRIST "TOTAL t OP STUDENTS IN ALL GROUPS" J 
SO INPUT H 

90 PRIST " NUMBER OP GROUPS"; 

100 IN£*0T X 

110 DIM C(K) 

120 PRINT 

130 POR I • 1 TO K 

135 PRINT 

140 A • A + 1 

150 PRINT "NUMBER IN GROUP ";Aj 

160 INPUT C 

170 PRINT 

180 POR T • 1 TO C 

150 PRINT "STUDENT ";T; 

200 INPUT X 

210 X2 • X A 2 

220 Tl « Tl + X2 

230 L(A) • L( A) + X 

250 NEXT T 

260 S2 » S2 + L(A) 

270 Q(R) - (L(A) A 2) / C 

280 S3 - S3 + Q(X) 

290 NEXT I 

300 SI - SI 4 Q(X) 

310 W » S2 A 2 / N 

320 Dl ■ X - J 

330 D3 - N - 1 

340 D2 - D3 - Dl 

350 PRINT 

360 PRINT 

370 PRINT "DF (BETWEEN, WITHIN, TOTAL)- "jDl^ * : 02 ; * , ";D3 

380 VI « (S3 - W) / Dl 

390 V2 - ((Tl - W) - (S3 - W) ) / D2 

400 P » VI / V2 

410 W2 - 1 - (((Tl - W) - (S3 - W)) / (Tl - K)> 
420 PRINT 
430 PRINT 

440 PRINT " F ■ ";F 

450 X - 1 

460 IP P < 1 THEN 510 

470 S - Dl 

480 T - D2 

490 2 • F 

500 GOTO 540 

510 S - D2 

520 T - Dl 

530 2 - 1 / P 

540 J - 2 / 9 / S 

550 X • 2 / 9 / T 

56*J Y - ABS ((1 - X) * 1 \ (1 / 3) - 1 + J) / SQR (X • 2 A (2 / 3 ) + J) 
570 IF T < 4 THEN 610 

580 X - .5 / (1 + Y * (.196854 + Y • (.115194 + t * (.000344 ♦ Y • .019527)))) 
. 590 X - INT (X * 10000 + .5) / 10000 
600 GOTO 630 

610 Y • Y • U > .06 • Y 4 / T A 3) 

620 GOTO 580 

630 IF F > -1 THEN 650 

640 X • 1 - X 

650 PRINT " P • ";X 

660 PRINT 
670 PRINT 

680 IF X < .05 THEN 710 

690 PRINT "NONE OF THESE GROUPS 

700 GOTO 720 

710 PRINT "AT LEAST ONE OF THESE GROUPS 
720 PRINT "HAS AN AVERAGE WHICH IS 

730 PRINT "(STATISTICALLY) SIGNIFICANTLY DIFFERENT 
740 PRINT "FROM THE TOTAL SAMPLE AVERAGE. 
750 END 
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110 
120 
130 
140 
150 
160 
1?0 
180 
190 
200 
210 
220 
230 
240 
250 
260 
270 
280 
290 
300 
310 
320 
330 
340 
350 
360 
370 
380 
390 
400 
410 
420 
430 
440 
450 
460 
470 
480 
490 
500 
510 
520 
530 
540 
550 

560 

570 

580 

590 

600 

610 

620 

630 

640 

650 

660 

670 

680 

690 

700 

710 

720 

730 

740 

750 

760 

770 
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C + J) 



mm 
print 

nntt "number of rows 

XJIPU? R 

PRINT "NUMBER OF COLUMNS 
I WOT C 
DM V1<R * C),V2(C),A(R) 
PRINT 

PRIST " CONTINGENCY TA) LEi ■ 

FOB I - 1 TO R 

PRINT "BOH ";I 

FOR J - 1 TO C 

PRIST 0 ELEMENT ";J; 

IHPUT VI ( (I - 1) 

■EXT J 

■EXT I 

PRINT 
M - 1 

FOR X - 1 TO R 

FOR J - 1 TO C 
A(I) - A(I) + VX(M) 
M - M + 1 

HEXT J 
L - L + A(I) 

NEXT I 
N - R * C 

FOR I « 1 TO C 

FOR J ■ I TO N STEP C 
V2(I) - V2(I) + V1(J) 

NEXT J 

NEXT I 

FOR I ■ 1 TO C 

FOR J « 1 TO R 
P « A(J) • V2(I) 
X « I + 

IF R < 

IF C < > 



DISTRIBUTION SIGNIFICANCE 



.5) A 2 / P 



(R - 1) 



/ L 
(J - 1) • C 
> 2 THEN 410 
2 THEN 410 

Y - ( ABS (VI (X) - P) 
GOTO 420 

Y - (Vl(X) - P) a 2 / P 
Z - Z + Y 

NEXT J 
NEXT I 
PRINT 

PRINT ■ CHI-SQUARE « ";2 

PRINT "DEGREES OF FREEDOM « "; (C - 1) 

V - (C - 1) • (R - 1) 

w ■ z 

R - 1 

FOR I ■ V TO 2 STEP - 2 
R - R * I 
NEXT * 

K - W A ( IMT ((V ♦ 1) / 2)) • EXP (-W/2J/R 

IF IN? (V / 2) - V / 2 THEN 580 
J - SQR (2 / W / 3.14159265) 

GOTO 590 
J - 1 
L - 1 
M - 1 

V ■ V + 2 
M - M • W / V 

IF M < .0000001 THEN 660 
M 



■;P 



L - L + 

GOTO 610 
P - 1 - J • R 
PRINT ■ 
PRINT 
PRINT 

PRINT "THESE OBSERVATIONS SUGGEST THAT 
PRINT "THE CATEGORIES 
IF P < .05 THEN 750 
PRINT "DO NOT AFFECT DISTRIBUTION 
GOTO 760 

PRINT "DO AFFECT DISTRIBUTION 
PRINT "WITH STATISTICAL SIGNIFICANCE. 
END 
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RELIABILITY — KUDER-RICHARDSOS 21 



10 

20 mm 

30 PRIST 
40 FRIST 

45 DIM A(100) 

46 DIM B(100) 

50 * **** "SOMBER OF SUBJECTS (TESTS) "i 
60 ISPUT S 
70 PRIST 

80 PRIST ■ NUMBER OP ITEMS PER TEST ■> 

81 ISPOT K 
90 PRIST 

100 PRIST "ENTER TEST SCORES" 

105 PRIST 

110 FOR I > 1 TO H 

120 PRIST "TEST "jlj 

130 INPUT A(I) 

140 T - T * A(I) 

150 NEXT I 

160 M ■ T / S 

170 *2R - 1ST (M * 100 + 0.5) / 100 

180 FOR I > 1 TO H 

190 FOR J ■ (I * 1) TO N 

200 IF A(I) > A I J ) THEN GOTO 240 

210 X - A(I> 

220 A(i) * 

230 hi J} m X 
NEXT J 

t2* i 1 TO N 
NEXT I 
PRINT 

FOR J » I TO K 
NEXT J 

FOR I - 1 TO » 
FOR J » 1 TO K 

IF A(l) - J THEN SU" 1 * B(J) + A 
NEXT J 
NEXT I 

PRINT "SCOR£","F"REQ* 
FOR J r* K TO 1 STEP - 1 

> 0 THEN PRINT u,i{J) 



240 

250 
260 
270 
280 
290 
300 
310 
315 
320 
330 
340 
350 
360 
370 
360 
390 
400 
410 
420 
430 



IF B(J) 
NEXT J 
FOR I - 
IF A(I) 
NEXT I 

IF C < > 0 THEN 
PGR I * 1 TO N 
440 X - ABS (A(I) - M) 
450 D - D + X A 2 
460 NEXT I 
470 V - D / N 
480 S - SQR (V) 
490 SR - 1ST (S 
500 Rl - (R * V) 
510 R2 - Rl / (V 
52X) RR - 1ST (R2 
530 PRIST 



TO N 

0 THEN C*CM 



PRINT "0",C 



540 PRINT 
550 PRINT 
560 PRINT 
570 PRI.NT 
580 PRINT 
590 SM - S 
600 SE - INT (SM 
61u PRINT 
620 END 



100 + 0.5) / 100 
(M * (K - M) ) 
(K - 1)) 

100 + 0.5) / 100 

MEAN SCORE - ";MR 
S.D. - -;SR 



K-R 21 RELIABILITY - a ;RR 

SQR (1 - R2) 

' 100 + 0.5) / 100 
STANDARD ERROR - (♦/-) 



B ;SSi 
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20 run - sigsxficasci of group mean 

22 9UR " COMPARED WITH GIVES MSAS 

30 nm 
35 ntxn 

40 PRIST "ESTER NUMBER IV GROUP 
50 ISPOT S 
70 PRIST 
190 E - 1 

200 FOR I - 1 TO SGN <B - 1) ♦ 1 

250 R(I) - »-* 

260 FOR J - 1 TO R(I) 

270 PRIST " CASE"; J; 

280 IVPDT P(J,I) 

290 V(X) - V(I) ♦ P(J,I) 

300 D(X) - D(I) + P(J,I) A 2 

310 EBXT J 

320 M(I) - V(I) / R(X) 

330 V(I) - (D(I) - V(I) A 2 / R( I ) ) / (R(I) - 1) 
340 HKXT I 
350 PRIST 

380 PRIST "VALUE OF MEAN (X)"; 
3*0 INPUT M 

400 A - (Md) • M) * SQR (R(l) / V(l)) 

410 B - R(l) - 1 

510 PRIST 

520 T - ABS (A) 

530 F - B 

700 PRINT 

710 PRINT " DF-";F 

720 X - 1 

730 * - 1 

740 T - T A 2 

750 IP T < 1 THEN 800 

760 S - Y 

770 El - F 

780 2 « T 

790 GOTO 830 

80C S - F 

810 El - Y 

eae % * i / t 

830 J ■» 2 / 3 / S 
840 Fl * 2 / 9 / Ei 

850 % ~ ABS ({1 FX) • S a <1 / 3) - 1 + J) / SQR (Fl * 2 a (2 / 3 ) + J) 
860 'ST El < 4 THEN $00 

870 X ■« »25 / (1 + L * (.196854 + L * (.115194 + L * (.000344 + L • .019527)))) a 
880 X IWT (X * 10000 + .5) / 10000 
890 GOTO *2Q 

900 L - L * (1 * oQ8 fc t A 4 / Ei a ?5 

910 GOTO 870 

920 IF T > « 1 TSFH §40 

930 X - 1 - X 

940 PRINT " P-";X 
950 PRINT 

960 PRINT "TBIS GROUP MEAN 
970 IF X < .05 THEN 1000 
980 PRINT "IS NOT 
990 GOTO 1010 
1000 PRINT "IS 

1010 PRINT "(STATISTICALLY) SIGNIFICANTLY Dl EFFERENT FROM TEE GIVEN MEAN 
1020 END 
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10 HOME 

20 PUR • TEST ITEM ANALYSIS 

30 PRINT 

40 PUR 

50 PRINT "HOW MAM SUBJECTS TOOK TEST "j 

co input N 

70 S « INT (V • .27) 

•0 PRINT 

85 PRINT 

90 PRINT •SELECT THE " ? S?' LOWEST SCORING TESTS" 
100 PRINT "AMD THE "jS;" HIGHEST SCORING TESTS 
110 PRINT 
120 PRIST 

130 PRINT "HOW MANY ITEMS WILL YOU ANALYZE "; 

140 INPUT M 

150 POR I - 1 TO M 

160 HOKE 

170 PRINT •HOW MANY CORRECT IN UPPER GROUP "; 
130 INPUT U 

190 PRINT "HOW MANY CORRECT IN LOWER GROUP m t 
200 INPUT L 
210 PRINT 

215 Dl ■ (U + L) / (S * 2) 

220 PRINT * INDEX OP DIPPICULTY » ";D1 

230 D2 « (U - L) / S 

240 PRINT ■ INDEX OP DISCRIMINATION ■ ";D2 

245 PRINT 

246 PRINT 

250 IP D2 > .39 THEN GOTO 300 

260 IP D2 .29 THEN GOTO 320 

270 IP D2 > .19 THEN GOTO 340 

280 PRINT "A POOR ITEM. REVISE OR DISCARD. 

290 GOTO 350 

300 PRINT "A VERY GOOD ITEM!" 

310 GOTO 350 

320 PRINT "REASONABLY GOOD. MAY BE IMPROVED" 

330 GOTO 3<»0 

340 PRINT "MARGINAL ITEM. NEEDS IMPROVEMENT." 

350 PRINT 

351 PRINT 

352 PRINT 

353 PRINT 
360 PRINT 
370 PRINT 
380 PRINT 

390 PRINT "PRESS ANY KiTY POR NEXT ITEM 

400 INPUT K$ 

410 NEXT I 

420 END 
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